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ABSTRACT 

To help vocational education administrators and 
practitioners r this paper presents a framework for vocational 
education program evaluation. Evaluation is defined as the act of 
rend^. ag judgments to determine a program's value? two major types 
are formative and summative. Vocational education evaluation should 
take a broader and more comprehensive approach than a focus on 
student achievement. This approach requires use of valid and reliable 
information of three interrelated types: needs , processes, and 
outcomes. The needs for vocational education include students 1 
interests, knowledge, attitudes, and skills? the knowledge, 
attitudes, and skills employers want in employees? and the 
educational, economic, and social needs of society. Sources of 
information about vocational education program processes include: (1) 
organizational information? (2) program information? (3) support 
services and activities? (4) staff? (5) student demographic 
characteristics and educational achievement? and (6) community 
information. Also part of the evaluation framework are economic 
outcomes (labor force participation, employment and unemployment 
rates, training-related placement, type of employment, earnings, 
employee satisfaction with work, employer satisfaction with 
employees)? educational outcomes (academic and higher-order thinking 
skills, knowledge of the world of work, occupational skills, school 
attendance and dropout rates, continuing education rates, 
satisfaction with education)? and psychosocial outcomes (aspirations, 
attitudes, values, self-esteem, citizenship, and leadership). (28 
references) (SK) 
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FOREWORD 
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that is funded by the Office of Educational Research and Improvement (OERI), U.S. 
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be of interest to vocational education administrators, practitioners, and evaluators engaged 
in assessing vocational education programs. 

ERIC/ACVE would like to thank N. L. McCaslin for his work in the preparation of this 
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the Evaluation and Policy Division of the National Center for Research in Vocational Edu- 
cation at Ohio State. He has served as an evaluation consultant in such varied settings as 
the U.S. Army, community colleges, universities, state departments of education, state advi- 
sory councils, the Unesco Institute of Education, and government ministries in Mexico, 
Uganda, and Scotland. 
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and Training for Employment. 

Publication development was coordinated by Susan Imel. Sandra Kerka edited the manu- 
script, and Janet Ray served as word processor operator. 
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EXECUTIVE SUMMARY 



One effect of educational reform movements is greater emphasis on evaluation. To help 
vocational education administrators and practitioners with program assessment, this paper 
presents a framework for vocational education program evaluation. Evaluation is defined 
as the act of rendering judgments to determine a program's value; two major types are 
formative and summative. Vocational education evaluation has been encouraged or 
mandated by the Vocational Education Act of 1963, the 1968 Amendments, the Educational 
Amendments of 1976, and Carl D. Perkins Vocational Education Act of 1984, and the 
recently enacted Carl D. Perkins Vocational and Applied Technology Education Act. 

Vocational education evaluation should take a broader and more comprehensive approach 
than a focus on student achievement. This approach requires use of valid and reliable 
information of three interrelated types: 

1. The needs for vocational education expressed by students, employers, and society 

2. Vocational education program processes 

3. Vocational education program outcomes 



A framework for evaluating vocational education consists of the following components: 



Needs 

1. Students' interests, knowledge, attitudes, and skills 

2. The knowledge, attitudes, and skills employers want in employees 

3. The educational, economic, and social needs of society 



Vocational Education Processes 

1. Organizational information (organizational arrangement, administrative style, organiza- 
tional climate, teaching styles, type of organization) 

2. Program information (goals and objectives, instructional content and delivery methods, 
administrative support, recruitment, facilities and equipment, cost) 
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3. Support services and activities information (student organizations, job placement, 
continuing education, planning, evaluation, personnel development) 

4. Staff information (demographic characteristics, educational experiences and competence, 
occupational experiences and competence) 

5. Student information (demographic characteristics, educational achievement) 

6. Community information (linkage with business, industry, other educational agencies, 
other training programs; type of community) 



Vocational Education Outcomes 

1. Economic outcomes (labor force participation, employment and unemployment rates, 
training-related placement, type of employment, earnings, employee satisfaction with 
work, employer satisfaction with employees) 

2. Educational outcomes (academic and higher-ordei thinking skills, knowledge of the 
world of work, occupational skills, school attendance and dropout rates, continuing 
education rates, satisfaction with education) 

3. Psychosocial outcomes (aspirations, attitudes, values, self-esteem, citizenship, leadership) 
Steps foi using the evaluation framework are as follows: 

• Determining the type, purpose, and extent of the evaluation 

• Selecting the components 

• Identifying the sources and types of information to be collected 

• Collecting the required information 

• Analyzing and interpreting the information 

• Preparing an evaluation report 

Information on vocational education program evaluation may be found in the ERIC system 
using the following descriptors: Accountability, Data Collection, Educational Change, 
•Evaluation Criteria, Information Needs, Information Sources, "Outcomes of Education, 
•Program Evaluation, Program Improvement, "Vocational Education. Asterisks indicate 
descriptors that are particularly relevant. 
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INTRODUCTION 



Evaluation has taken on increased mean- 
ing and importance in the last decade with 
the emphasis that has been placed on 
educational reform. The educational 
reform movement first became apparent 
when the Secretary of Education T. H. 
Bell, created the National Commission on 
Excellence in Education on August 26, 
1981, and charged it to report on the 
quality of education in the United States. 
The commission report, -4 Nation At Risk 
The Imperative for Educational Reform 
(NCEE 1983) stated: 

If an unfriendly foreign power 
had attempted to impose on 
America the mediocre educa- 
tional performance that exists 
today, we might well have viewed 
it as an act of war. (p. 5) 

This report stimulated interest throughout 
the country and many states initiated 
similar study groups. These groups were 
asked to examine their educational sys- 
tem, determine problems, and recommend 
changes for improving their programs. 

In the 1990s there continues to be wide 
concern over the level of educational 
achievement in the United States. 
According to The 22nd Annual Gallup 
Poll of the Publics Attitudes toward the 
Public Schools 11 (Elam 1990), more than 
three-quarters of those questioned indi- 
cated that they attach very high or high 
priority to all six of the national goals for 
education proposed by President Bush 
and the 50 state governors. President 
Bush, in his concluding remarks to the 
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Education Summit he*d at the University 
of Virginia on September 28, 1990, stated: 

There is a need for the first time 
in this nation's history to have 
specific results-oriented goals (for 
education). We recognize the 
need for . . . accountability. 

The National Governors' Association 
(1990) stated that the country is facing a 
major crisis in education. They reported 
that the educational system is obsolete 
due to technological, economic, social, 
and demographic changes. According to 
their report- 
over the course of the next dec- 
ade, our nation must better edu- 
cate far more Americans, of all 
ages, to new kinds and higher 
levels of knowledge and skills 
than ever before. We must do 
this with an increasingly diverse 
population, many of whom face 
substantial economic, social or 
other barriers to learning, such as 
the effects of substance abuse, 
teen pregnancy or inadequate 
health care. None of our citizens 
can be written off. (p. 7) 

Several states (for example, California, 
Connecticut, Iowa, Kansas, Maryland, 
New Jersey, and Ohio) have taken steps 
to make their educational systems more 
accountable for their outcomes. These 
include such outcomes as basic skill 
achievement, attendance, and dropout 
rates. Regardless of the outcomes select- 
ed, the message is clear: states want their 



schools to demonstrate a resp .asible level 
of accountability for the funds that have 
been entrusted to them. 

If educational programs are to reach their 
objectives, increased levels of cooperation 
among the schools, parents and students, 
and business and industry are required. 
Much of this effort in the past has been 
fragmented with little articulation of 
effort. Much more needs to be done to 
smooth the transitions between education 
and work. 

Although change ; s necessary, educators, 
parents, and employers often find it dif- 
ficult. All too often, we are uncomfort- 
able with the adjustments it requires and 
want to hold on to the practices of the 
past. This type of behavior must be dis- 
carded as we approach the 21st Century. 



The field of vocational education has not 
been unaffected by the educational reform 
movement for it has also spurred interest 
in the evaluation of vocational education 
programs. This paper is intended to help 
administrators and vocational education 
practitioners with the evaluation of local 
vocational education programs. It begins 
by defining evaluation and then provides 
an overview of evaluation in vocational 
education. Next, types of information 
needed for evaluating vocational educa- 
tion are described. Then, a framework 
for evaluating vocational education, which 
uses these information categories as a 
basis, is presented. The paper concludes 
with recommendations for using the eval- 
uation framework. 



EVALUATION IN VOCATIONAL EDUCATION 



One of the major problems in discussing 
evaluation is that there is no widely 
accepted definition of evaluation. Indeed, 
this lack of agreement has led to the 
development of a variety of approaches to 
evaluation. Worth °.n and Sanders (1987) 
classified the different approaches into the 
following six categories: 

1. Objectives-oriented approaches 

2. Management-oriented approaches 

3. Consumer-oriented approaches 

4. Expertise-oriented approaches 

5. Adversary-oriented approaches 

6. Naturalistic and participant-oriented 
approaches 

For the purposes of this paper, the defi- 
nition of evaluation offered by Worthen 
and Sanders (1987) is used: "evaluation 
. . . (is) the act of rendering judgments to 
determine (a program's) value -worth and 
merit" (p. 24). 

Two major types of evaluation are gener- 
ally accepted in discussions of evaluation: 
formative and summative. Michael 
Scriven (1967) was the first to differenti- 
ate between these two types of evaluation. 

Formative evaluation is conducted 
throughout a program to improve its oper- 
ation. This type of evaluation primarily is 
designed to provide feedback for revising 



or improving the procedures used in the 
programs. 

Summative evaluation, on the other hand, 
is conducted at the conclusion of a pro- 
gram to provide evidence of its worth or 
merit. This type of information is often 
used to document the effectiveness of a 
program. In this sense, the purpose of 
summative evaluation is accountability. 

Evaluation has been an area of concern 
for vocational education teachers and 
administrators for many years. Early eval- 
uation efforts centered on conducting 
follow-up studies to determine the degree 
to which vocational education graduates 
found placements in jobs that were related 
to their training. This type of evaluation 
is summative. Other early forms of evalu- 
ation relied on an accreditation type of 
visit by external teams of individuals. 
This is an example of formative evalua- 
tion. 

The 1963 Vocational Education Act was 
one of the first pieces of federal legis- 
lation indicating the need for states to 
conduct evaluations of their vocational 
education programs. Many evaluation ef- 
forts resulting from this mandate consisted 
of reviewing vocational education proces- 
ses and follow-up information on program 
graduates. 

In 1968, the Vocational Education 
Amendments to the 1963 Vocational Edu- 
cation Act continued to emphasize state 
evaluation activities. They also gave 
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evaluation responsibilities to state advisory 
councils. 

The Education Amendments of 1976 (P.L 
94-482), which revised the Vocational 
Education Act of 1963, further expanded 
the responsibility for evaluation and had 
at least 28 references to different forms of 
evaluation (Wentling 1980). These evalu- 
ation requirements were designed to in- 
crease the responsiveness of vocational 
education to changing labor markets, and 
they required states to evaluate their 
programs every 5 years and to determine 
the extent to which programs completers 
and leavers- 

• found employment in occupations re- 
lated to their training, and 

• were considered by their employers to 
be w-Ml-trained and prepared for 
employment. 

Seven different groups were specifically 
charged with conducting vocational educa- 
tion evaluations in the 1976 Amendments: 

1. State boards of vocational education 

2. State advisory councils for vocational 
education 

3. The National Advisory Council for 
Vocational Education 

4. The U.S. Office of Education's 
Bureau of Occupational and Adult 
Education 

5. The National Institute of Education 

6. The National Center for Education 
Statistics 

7. The U.S. Office of Education's Office 
of Evaluation and Dissemination 



Ihe Carl D. Perkins Vocational Education 
Act (P.L 98-524) was enacted in 1984. 
This legislation was designed- 

to amend the Vocational Educa- 
tion Act of 1963 to strengthen 
and expand the economic base of 
the Nation, develop human re- 
sources, reduce structural unem- 
ployment, increase productivity, 
and strengthen the Nation's de- 
fense capabilities by assisting the 
States to expand, improve, and 
update high-quality programs of 
vocational-technical education, 
and for other purposes, (p. 2435) 

The evaluation requirements of P.L. 98- 
524 charged the states to develop mea- 
sures for the effectiveness of vocational 
education programs including the follow- 
ing: 

• The occupations to be trained for, 
which will reflect a realistic 
assessment of the labor market of 
the state; 

• The levels of skills to be achieved 
in particular occupations, wiiich 
will reflect the hiring needs of 
employers; and 

• The basic employment competen- 
cies to be used in performance 
outcomes, which will reflect the 
hiring needs of employers, 
(p. 2447) 

Congress has recently enacted the Carl D. 
Perkins Vocational and Applied Technol- 
ogy Education Act of 1990. This legisla- 
tion continues to include evaluation spe- 
cifications. It requires states to develop 
core standards and measures of perfor- 
mance for secondary and postsecondary 
vocational education programs. These 
performance standards are to include the 
following: 



INFORMATION NEEDED FOR VOCATIONAL 
EDUCATION EVALUATION 



If vocational education is to provide the 
citizens of the United States with the 
skills they need to compete in the market- 
place and lead meaningful lives, then the 
evaluation of these programs becomes 
even more important. Much of the cur- 
rent literature on educational reform 
could lead one to conclude that the only 
important evaluation information is 
related to results in terms of student 
achievement. However, if information on 
student achievement in vocational and 
roademic skills is all that is available, how 
can teachers and administrators know 
what worked and did not work? Upon 
what basis should decisions to add and 
drop programs be made? What should be 
changed in order for vocational students 
to achieve at higher levels? 

This paper suggests that those concerned 
with evaluating vocational education con- 



sider a broader and more comprehensive 
approach to evaluation. This approach 
requires the use of valid and reliable 
information of three major types: 
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The needs for vocational education as 
expressed by clients (that is, students 
and employers) and society 

The educational processes followed 
by vocational education programs 

The outcomes achieved by vocational 
education programs 



The interrelationships of these three types 
of evaluation information are presented in 
figure 1. A brief description of these 
three types follows. 



Needs for Vocational Education 



Vocational 

Education 

Processes 



Vocational 

Education 

Outcomes 



Figure 1. Interrelationships of evaluation information 



Needs for Vocational Education 

Vocational education programs in the 
United States are basically offered as elec- 
tees in secondary and postsecondary edu- 
cation agencies. Therefore, individuals 
will not enroll unless they feel that a 
program would meet their ueeds or satisfy 
their desire for a type of service they 
would like to receive. Thus, vocational 
education is a "customer" driven program, 
"Customers" here are broadly defined to 
include all groups in a community that 
might have a stake in vocational education 
(Rossi and Freeman 1989). 

Kaufman and Stone (1983) indicated that 
in assessing needs it is necessary to show 
and document gaps between current re- 
sults and desired results (in terms of 
outcomes) and then to place these gaps 
(needs) in priority order. Accordingly, 
needs are concerned with determining the 
extent or magnitude of a problem. 

Needs can be determined by reviewing ex- 
isting information (for example, labor 
market information and educational 
achievement levels) or collecting new 
information. This information might be of 
either a quantitative or qualitative nature. 



Vocational Education Processes 

Historically, accreditation reviews have 
devoted much of their attention to exam- 
ining the processes used by educational 
agencies. Many states' 5-year reviews of 
their vocational education programs have 
relied on this type of information. Pro- 
gram sponsors/funders typically place 
heavy emphasis on process information in 
demonstrating their accountability. 



Process information provides evaluators 
with a basis for understanding and inter- 
preting what has occurred. If vocational 
education programs are to be improved as 
a result of an evaluation, it is essential to 
provide information about how they have 
been conducted. Rossi and Freeman 
(1989) referred to the evaluation of pro- 
cesses as systematic monitoring to exam- 
ine program coverage and delivery. 



Vocational Education Outcomes 

As previously discussed, the current 
emphasis in educational evaluation is on 
documenting educational effectiveness in 
terms of student achievement. Stuffle- 
beam (1969, 1971, 1983) and Stufflebeam 
and Shinkfield (1985) referred to evalu- 
ating the outcomes of educational pro- 
grams as product evaluation. Among the 
questions th?t product evaluation answers 
are the following: 

• What results were obtained? 

• How well were needs reduced? 

• What should be done with the pro- 
gram after it has run its course? 

Rossi and Freeman (1989) referred to 
evaluating outcomes as "impact assess- 
ments (which) are undertaken to establish 
whether or not interventions produce their 
intended effects" (p. 225). The National 
Governors' Association (1990) report, 
Educating America: State Strategies for 
Achieving the National Education Goals, 
stated that "national education goals will 
be meaningless unless progress toward 
meeting them is measured accurately and 
adequately, and reported to the American 
people" (p. 39). 
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A FRAMEWORK FOR EVALUATING 
VOCATIONAL EDUCATION 



Using the three types of information need- 
ed for evaluating vocational education, an 
evaluation framework is presented in fig- 
ure 2. This framework is not proposed as 
a rigid model that all evaluations should 
follow. Rather, it is presented as a guide 
for carefully thinking through what might 
be included in an evaluation of vocational 
education. The remainder of this section 
discusses possible components for each of 
the three types of information. 



The Needs of Students, Employers, 
and Society 

In the Peters and Waterman (1987) book, 
In Search of Excellence, Lew Young, 
Editor-in-Chief of Business Week, is 
quoted as saying: 

Probably the most important 
management fundamental that is 
being ignored today is staying 
close to the customer to satisfy 
his (sic) needs and anticipate his 
(sic) wants, (p. 156) 

Educators need to understand and antici- 
pate the needs of their customers. If the 
needs of the vocational education custo- 
mer are to be taken seriously, educators 
must begin collecting better information 
about students, employers, and society. 



Students 

Today's vocational education students 
come with a variety of needs. Increas- 
ingly, they come from one-parent, lower 
socioeconomic status, and minority fami- 
lies. The Forgotten Half: Pathways to 
Success for America's Youth and Young 
Families (The William T. Grant Founda- 
tion Commission on Work, Family and 
Citizenship 1988) reported that- 

our two-year study of 16-24-year- 
olds has convinced us that, as 
young Americans navigate . . . 
from youth to adulthood, far too 
many flounder and . . . fail. . . . 
Although rich in material re- 
sources, our society seems unable 
to ensure that all our youth will 
. . . face their futures with . . . 
confidence and security. This is 
especially true of the 20 million 
non-college-bound young people 
we have termed the forgotten 
half. (p. 1) 

Similarly, vocational educators must be- 
come increasingly concerned with adults 
who are struggling to survive in the work- 
place. Of particular concern among the 
potentuJ students in these categories are 
disadvantaged or handicapped people, 
people entering nontraditional occupa- 
tions, homemakers, immigrants, individu- 
als with limited English proficiency, and 
individuals who are incarcerated. The 
workplace is demanding that even those 
with "good jobs" receive training and 
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OUTCOMES 



Economic 
Labor Force Participation Rates 
Employment and Unemployment Rates 
Training-Related placement 
Type of Employment 
Earnings 

Employee Satisfaction with Work 
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Educational 
Academic Skills 
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Continuing Education Rates 
Satisfaction with Education 
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Aspirations 
Attitudes and Values 
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Figure 2. A framework for evaluating vocational education 
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retraining to meet the rapidly changing 
demands of the global market. 

In attempting to determine the needs of 
these vocational education customers, in- 
formation is needed regarding their inter- 
ests, knowledge, attitudes, and skills. 
Without taking this type of information 
into consideration, vocational education 
programs are likely to be judged irrelevant 
by these individuals. As a result, they will 
seek education and training from other 
agencies or become dropouts-further re- 
ducing the nation's productivity and eco- 
nomic growth. 



Employers 

Another important group of customers for 
vocational education are the employers. 
The chief executive officers (CEOs) of 
large businesses have become more vocal 
in expressing the needs of their organiza- 
tions (Iacocca and Novak 1987; Kearns 
and Doyle 1988; Sculley and Byrne 1987). 
However, these businesses are only one 
type of vocational education employer cus- 
tomer. Their needs may not be the same 
as those of other employer customers--the 
medium-sized and small businesses who 
also employ vocational education students 
or the entrepreneurs who create their own 
jobs after completing a vocational educa- 
tion program. 

Cetron and Davies (1989) projected that 
"by the turn of the century, most of our 
mid-sized institutions will have vanished, 
but thousands of tiny companies will be 
flourishing beneath the feet-and notice- 
of the giants" (p. 204). Whether or not 
this speculation comes about, vocational 
educators need to consider who their em- 
ployer customers are. Regardless of 
whether these customers are large, medi- 
um, or small, educators need to seek their 
advice regarding the knowledge, attitudes, 



and skills they would like in their 
employees. 

It is also important to note that, in the 
large and medium-sized businesses, input 
should be sought from more than one in- 
dividual. This is particularly important 
because businesses are increasing the 
amount of employee participation in man- 
agement and stressing the autonomy and 
creativity of the worker. Information on 
needs collected from such individuals as 
personnel directors, division heads, and 
line supervisors is essential in determining 
the knowledge, attitudes, and skills needed 
by these employer customers. 

The needs of the entrepreneur as an em- 
ployer customer have been overlooked in 
most evaluation efforts in vocational edu- 
cation. If Cetron and Davies (U69) are 
correct in their prediction about the dra- 
matic increase in small companies, the 
needs of this type of customer must be 
determined. 



Society 

Another important customer of vocational 
education is society itself. These custo- 
mers include such groups as taxpayers, 
school board members, boards of trustees, 
legislators, policy makers, and parents. 
The educational, economic, and social 
needs of society are often expressed by 
the media or through statistical and ana- 
lytical reports published by the govern- 
ment and private foundations. 

Reports on the educational needs of soci- 
ety are particularly abundant in the media. 
Among the types of information express- 
ing educational needs are school census 
data, schtol attendance and dropout rates, 
basic education achievement scores, and 
rates of continuing postsecondary educa- 
tion. 
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A variety of economic information and 
statistics is published at the national, state, 
and local levels. These figures provide 
information that is relevant to those con- 
cerned about vocational education pro- 
grams. Among the types of economic in- 
formation are unemployment rates, num- 
ber of building permits issued, average 
hours employees work per week, and 
labor market projections. 

A number of social indicators are also rel- 
evant to vocational education programs. 
Among the types of information that 
could be used are census information 
(particularly since the 1990 information 
was jubt collected) on numbers of school- 
aged students, racial composition, and 
marital status. Statistics on the number of 
teenaged parents, percentage on public 
assistance, number receiving Aid to 
Families with Dependent Children, 
number of handicapped individuals, and 
number receiving free or reduced price 
school lunches represent yet another 
source of valuable information. 



Vocational Education Processes 

An examination of various processes of 
vocational education has been a major 
focus of statewide evaluation efforts since 
the implementation of the 1968 Amend- 
ments to the Vocational Education Act of 
1963. Building upon the model used by 
accreditation organizations, these efforts 
relied heavily upon a process evaluation in 
which review teams went into the schools, 
following a self-study, to interview, ob- 
serve, and review documents related to 
how these programs were operated. 
These statewide evaluation systems often 
included specific components such as cur- 
riculum and instruction, facilities, equip- 
ment, staff, student assessment, and stu- 
dent placement and follow-up. Many 
states continue to use these same types of 
statewide evaluations today and have 



expanded the process to include other 
components such as those dealing with 
equity, disadvantaged, handicapped, and 
nontraditional enrollments. 

In preparing this paper, the author re- 
viewed materials from many statewide 
evaluation systems and other vocational 
education evaluation studies. As a result, 
six major types of information emerged as 
important in conducting process evalua- 
tions of vocational education: organiza- 
tional information, program information, 
support services and activities information, 
staff information, student information, and 
community information. 



Organizational Information 

Public vocational education programs op- 
erate within the broader environment of 
a secondary or postsecondary educational 
organization. How these organizations 
behave and operate is an important 
source of information in evaluating voca- 
tional education programs. Information 
about an organization can be obtained by 
interviews, observations, reviews of exist- 
ing documents and information, and sur- 
veys-either individually or in combination. 
Potential areas of interest in an evaluation 
include the following: 



• Organizational arrangements 

• Administrative style 

• Organizational climate 

• Teaching styles 

• Type of educational organization 



Two important considerations in evaluat- 
ing the organizational arrangements for 
vocational education are its administrative 
structure and physical location. The or- 
ganizational chart provides an initial i 1i- 
cation of how the school or college *s 
structured to administer vocational educa- 
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tion programs. However, it is often infor- 
mative to discuss the administrative ar- 
rangements with faculty and administra- 
tors to find out how close the stated 
administrative arrangements are to the 
actual situation. Another item to consider 
is the program's physical location within a 
building. The accessibility of a program 
to students can be an important factor in 
determining who enrolls. 

Administrators are often identified as a 
key factor in having an effective organiza- 
tion. These individuals usually set the 
tone and direction for the organization. 
How administrators perceive their role in 
carrying out their functions (administrative 
style) serves as an important signal to fac- 
ulty and students in determining the ex- 
tent to which they will take responsibility 
in the teaching and learning process. 

Another important characteristic to be 
considered is that of organizational cli- 
mate—the set of properties that are per- 
ceived by the administrators, faculty, and 
students as influencing the performance of 
people in an organization (adapted from 
Gibson, Ivancevich, and Donnelly 1979). 
Therefore, organizational climate is an 
important element in helping to shape the 
opinions of and anticipated outcomes for 
vocational education programs. 

The teaching styles used in an educational 
organization are also important considera- 
tions in evaluating vocational education 
programs. One of the hallmarks of voca- 
tional education has been that of learning 
by doing. This educational process re- 
quire students to become active in con- 
crete learning situations, whether they be 
developing basic educational skills or 
occupational skills. An evaluation of 
vocational education should examine the 
appropriateness and relevance of the 
teaching styles to the outcomes expected. 



The type of agency offering vocational 
education programs is often overlooked 
when evaluations of these programs are 
conducted. Not all secondary and post- 
secondary agencies are alike. Secondary 
schools can be classified into organizations 
such a* comprehensive high schools con- 
taining their own vocational education 
programs, high schools that cooperate 
with other schools in offering vocational 
education programs at another location, 
and separate vocational education high 
schools with their own academic offerings. 
Similarly, postsecondary schools can be 
classified as community colleges, technical 
colleges/institutes, or technical colleges/ 
institutes within 4-year colleges. These 
types of organizations often have decided- 
ly different types o f vocational programs. 
In other instances, one educational organi- 
zation might ctfer both secondary and 
postsecondary programs. Therefore, the 
type of educational organizations offering 
vocational education programs should be 
identified so that the results of the eval- 
uation could be compared with those of 
similar organizations. 



Program Information 

Information related to how vocational 
education programs are conducted is im- 
portant in determining what changes need 
to be made in these programs or in deter- 
mining how to implement similar pro- 
grams in another location. The degree to 
which programs are operated as they were 
designed is essential if one is to judge the 
outcomes obtained. Potential areas of 
interest include- 
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• 


Goals and objectives 


• 


Instructional content 


• 


Instructional delivery methods 


• 


Administrative support 


• 


Recruitment 


• 


Facilities and equipment 


• 


Cost 



The goals and objectives provide the basic 
structure and road map for vocational 
education programs. Clear statements of 
goals and objectives are essential if every- 
one is to develop a sense of agreement 
about the purposes of these programs. 
These statements should be included in 
documents such as curriculum guides, 
courses of studies, or plans of instruction. 

The instructional content should be con- 
sistent with the goals and objectives. This 
content should be based on current indus- 
try standards and should reflect the needs 
of students and the community. Many vo- 
cational education programs have deter- 
mined the instructional content by tech- 
niques such as task analyses and DACUM 
(Developing A Curriculum). The curricu- 
lum guides, courses of studies, or plans of 
instruction should also be examined to 
determine the scope and sequence of the 
instruction and see how safety instruction 
is provided. 

The instructional delivery methods should 
be well-balanced, with a variety of teach- 
ing and learning approaches in the class- 
room instruction, supervised experience, 
and vocational student organizations. Evi- 
dence related to how articulation and 
integration of vocational and academic 
instruction are achieved should also be 
obtained. 

Administrative support should not be 
overlooked as an area of interest. On the 
surface this area may appear to be easy to 
determine. In making determinations of 



administrative support for vocational edu- 
cation, the perceptions of several different 
stakeholders need to be checked as well 
as reviewing advisory committee minutes 
and budget allocations. 

Recruitment of students is especially 
important for vocational education since 
these programs are elective. Teachers, 
administrators, and guidance counselors 
will need to have recruitment and promo- 
tional materials available for students 
considering taking vocational education 
programs or courses. The content of 
these materials should be reviewed with 
both accuracy and equity concerns in 
mind. 

Facilities and equipment represent yet 
another area of interest related to pro- 
gram information. Just as students in the 
science areas need access to up-to-date 
facilities and equipment, vocational educa- 
tion students also require access to mod- 
ern technology. The provision of facilities 
and equipment may be a shared responsi- 
bility of both the school and business and 
industry. Some skills needed in the oper- 
ation of equipment might be developed 
during the classroom instruction and other 
required skills may need to be developed 
during the student's supervised occupa- 
tional experience program. 

Finally, the cost of the program should be 
reviewed. A major item will be personnel 
costs. However, budgets should be exam- 
ined to see that additional funds are 
provided for items such as instructional 
materials, equipment, and supplies. 



Support Services and Activities 
Information 

To offer an effective program of voca- 
tional education, several support services 
and activities are recommended. An 
evaluation of vocational education needs 
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to consider the degree to which these 
services and activities are provided. 
Potential areas of interest include- 



• Student organizations 

• Job placement service 

• Continuing education service 

• Planning 

• Evaluation 

• Personnel development 



Student organizations have been an 
important aspect of vocational education 
for many years. These organizations 
include the Distributive Education Clubs 
of America (DECA), FFA (formerly the 
Future Farmers of America), Future Busi- 
ness Leaders of America (FBLA), Future 
Homemakers of America (FHA), Health 
Occupations Students of America 
(HOSA), Home Economics Related Occu- 
pations (HERO), Phi Beta Lambda 
(PBL), Vocational Industrial Clubs of 
America (VICA), and Technology Stu- 
dents Association (TSA). These organi- 
zations provide many opportunities for 
students to develop self-esteem, leader- 
ship, and citizenship. Additionally, these 
student organizations sponsor numerous 
competitive events and contests where 
their members' competence is evaluated. 

A job placement service is often included 
in vocational education programs to help 
students find employment. Individual 
teachers often take the sole responsibility 
for the placement of their students. At 
other times, especially in larger programs, 
the job placement service is centralized 
with individuals specifically responsible for 
this service. Some schools have job place- 
ment services using some centralized 
placement functions, but leaving the ac- 
tual responsibility of job placement with 
the teachers. 



A continuing education service is the 
assistance offered to students desiring to 
pursue additional schooling. It is as 
important to vocational education students 
as it is to academic students since educa- 
tion is increasingly viewed as a lifelong 
process. The increasing number of em- 
ployers requesting individuals with more 
than a high school education also rein- 
forces the need for this service. 

The necessity of planning from both a 
strategic and tactical point of view is 
being recognized as an essential compon- 
ent in reforming education. Ohio's Future 
at Work (Ohio Department of Education 
1990) is one example of the importance 
tha f state departments of vocational edu- 
cation are placing on developing strategic 
plans for the reform of education. These 
types of long-range plans develop a 3- to 
5-year vision for vocational education and 
provide the basis for tactical (operational) 
plans regarding what and how it will be 
carried out over each of the ensuing years. 

Evaluation activities are closely linked 
with planning. In planning, emphasis is 
placed on stating intentions ("What do we 
want to do?") whereas evaluation places 
emphasis on determining the degree to 
which stated expectations were met ("Did 
we do what we planned?"). Evaluation 
should examine the outcomes of a pro- 
gram and relate them to the various needs 
and processes contributing to or hindering 
these accomplishments so that the pro- 
gram can be improved. 

Personnel development encompasses those 
inservice activities primarily designed to 
develop or enhance the knowledge or 
skills of vocational education teachers and 
administrators. These activities can be of 
a formal or informal nature and offered 
either for credit or noncredit. The cur- 
rent educational reform movement will 
generate many reeds for personnel 
development. 
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Staff Information 

The teachers and administrators are the 
keystone in developing high quality voca- 
tional education programs. Efforts to 
evaluate vocational education need to 
consider information about their back- 
ground and experience. The types of 
information needed include- 



• Demographic characteristics 

• Educational experiences 

• Educational competence 

• Occupational experiences 

• Occupational competence 



Demographic information is important in 
dealing with issues such as the extent to 
which vocational education is dealing with 
equity issues and nontraditional programs. 
In these cases it would be important to 
know both the gender and race of the 
vocational education teachers and 
administrators. 

The educational experiences of vocational 
education teachers and administrators are 
quite varied. Sometimes they come di- 
rectly from business and industry with 
little or no formal postsecondary educa- 
tion. Some will have completed a 2-year 
program in a community college or tech- 
nical institute. Others will have com- 
pleted 4-year college degrees-some in 
education and some in other areas 
(health, business, agriculture, human 
ecology, engineering, and so forth). 
Similarly, these individuals have varied 
experiences in and with vocational educa- 
tion. Some will have worked only as 
teachers, others will have worked as both 
administrators and teachers. The type 
and amount of educational experiences 
that teachers and administrators possess 
represent impoi tant information in evalu- 
ating vocational education programs. 



Serious questions regarding the educa- 
tional competence of educators have been 
raised by many in the educational reform 
movement. To address these concerns, 
information is needed regarding the basic 
skill levels that vocational educators 
possess in reading, writing, and computa- 
tion. 

The amount of occupational experience 
that vocational educators possess is often 
included in evaluating vocational educa- 
tion programs. The recency of this experi- 
ence has been receiving increased atten- 
tion because of the rapid changes that are 
occurring in the workplace and society. 

Occupational competence represents yet 
another important source of information 
about the vocational education staff. The 
teachers* level of competence will affect 
their students' acquisition of the funda- 
mental skills necessary to become produc- 
tive members of the work force. Some 
states require occupational competency 
testing before a teacher can be certified. 
The educational reform concern over the 
degree to which teachers are competent to 
teach their subject matter is as important 
in vocational education as it is to the rest 
of the educational community. 



Student Information 

The need for information regarding the 
students whom vocational education serves 
is of paramount importance in evaluation. 
This information can be used initia^y as 
baseline information in determining what 
rtudents learn in vocational education. 
Furthermore, it can be used as a controll- 
ing variable when comparisons of achieve- 
ment with other types of students are 
made. Among the types of information 
needed are- 
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• Demographic characteristics 

• Educational achievement 



Students' demographic characteristics such 
as grade level, gender, race, and socio- 
economic status (SES) are valuable in 
understanding what type of student is 
participating in vocational education. 
Vocational education often has been view- 
ed as a second-rate education involving 
greater proportions of minorities and stu- 
dents with low SES. Vocational education 
has also been cLarged with sex-role ster- 
eotyping. Any evaluation efforts address- 
ing these concerns would need to examine 
student demographic information. 

The educational achievement in both aca- 
demic and vocational skills is especially 
important as more emphasis is being 
placed on the results of educational pro- 
grams. The present accountability move- 
ment has placed most of its emphasis on 
student achievement at the end of a 
course or program. However, approaches 
such as Value-added" evaluations, in 
which educational agencies need to docu- 
ment how much students learned, also will 
require educational achievement levels for 
students entering the program. 



Community Information 

The community has always played an im- 
portant role in vocational education. 
Since its inception, vocational educators 
have promoted the involvement of em- 
ployers and parents in planning and oper- 
ating their programs. The use of advisory 
groups to review a program, particularly 
to keep it up to date and relevant, has 
be*n a common practice. The Carl D. 
Perkins Vocational and Applied Technol- 
ogy Education Act of 1990 continues this 
tradition by calling for the involvement of 
employers, parents, teachers, students, and 



other providers in the planning, operation, 
and evaluation of vocational education 
programs. The types of information use- 

/ltd in the evaluation of these programs 

include- 



• Linkage with business and industry 

• linkage with other educational 
agencies 

• linkage with other training 
programs 

• Type of community 



A program's linkage with business and 
industry is not a recent development. 
These types of arrangements have been 
common since the inception of vocational 
education. One important use of advisory 
councils-consisting of personnel from 
business, industry, and labor-is for eval- 
uation. These groups are particularly well 
suited to determine needs for programs; 
identify and examine course content; re- 
view facilities, equipment and supplies; 
suggest procedures for coordinating and 
collaborating with business and industry; 
and assess student occupational compe- 
tence. 

The emphasis on linkage with other edu- 
cational agencies, on the other hand, is a 
more recent occurrence. The increasing 
competition for the limited funds available 
for supporting public education and train- 
ing programs has reinforced the need for 
these agencies and organizations to devel- 
op collaborative arrangements with one 
another. Many secondary and postsecond- 
ary agencies have developed articulation 
agreements to help students move from 
one program to another. 

Another more recent development is the 
linkage with other education and training 
programs* Not only do these arrange- 
ments often result in economic benefits, 
they also facilitate the delivery of services 
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to the participants. The 1990 Perkins Act 
permits states to use funds to coordinate 
services with the Job Training Partnership 
Act (JTPA) and the Wagner-Peyser Act 
for coordinated services. 

The type of community is another cate- 
gory of information that needs to be 
considered when conducting evaluations of 
vocational education. The various types 
of communities include rural, urban, and 
suburban areas. This identification will 
facilitate the comparison of program re- 
sults with similar communities, develop- 
ment of appropriate conclusions, and 
preparation of realistic policy recommen- 
dations. 



Vocational Education Outcomes 

Vocational education outcomes refer to 
the results or accomplishments of those 
who participate in these programs. These 
outcomes should be easily understood and 
agreed upon by vocational education 
stakeholders. An outcome might consist 
of either a single statistic or a composite 
measure based on several factors. The 
terms core standards, measures of perfor- 
mance, and performance standards are 
sometimes used in reference to educa- 
tional outcomes (as in the Carl D. Perkins 
Vocational and Applied Technology Edu- 
cation Act of 1990 and the Job Training 
Partnership Act). 

Tnc outcomes of vocational education 
often have been thought of solely in 
economic terms. However, more recent 
discussions and studies have included a 
wider range of educational, psychological, 
and sociological outcomes. For the pur- 
poses of this paper, three major types of 
outcomes are suggested: economic, edu- 
cational, and psychosocial. 
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Economic Outcomes 

As mentioned earlier, much of the re- 
search on the effectiveness of vocational 
education has focussed on economic out- 
comes. These types of outcomes are us- 
ually measured in terms of the experi- 
ences of former vocational education stu- 
dents in the labor market. Examples of 
economic outcomes include- 



• Labor force participation rates 

• Employment and unemployment 
/ates 

• Training-related placement 

• 1;/* of employment 

• Earnic 

• Employee satisfaction with work 

• Employer satisfaction with 
employees 



Labor force participation rates refer to 
the extent to which individuals are actu- 
ally working or seeking work. It does not 
include those individuals who have "given 
up" and stopped seeking work even 
though they are unemployed. 

The outcomes of employment and unem- 
ployment rates are determined by the pe 
centage of individuals who are either 
working or not working. This outcome, as 
well as the previous one, can be compared 
with state and national statistics for simi- 
lar groups of individuals if the same defi- 
nitions have been used in computing the 
rates. 

Historically, training-related placement 
has been the major outcome for evalua- 
ting vocational education. This outcome 
refers to the percentage of individuals 
who find employment in the occupations 
for which they were trained. This out- 
come often has been criticized since its 
achievement is determined by many fac- 
tors that are beyond the control of voca- 



tional educators. However, it has remain- 
ed an important outcome. 

The type of employment has also become 
an important outcome as emphasis on vo- 
cational education's role in economic 
development has increased. Since many 
new jobs are in small businesses, self- 
employment has become a legitimate out- 
come for vocational education, and many 
entrepreneurship programs have been de- 
veloped with this specific result in mind. 
The percentage of males and females dis- 
tributed across all types of occupations is 
another important factor, and niany pro- 
grams have beer specifically designed to 
address inequities in these occupational 
areas. 

Earnings received from working represent 
another type of outcome for vocational 
education programs. In determining earn- 
ings, it usually helpful to think in terms of 
annual earnings that reflect both the hour- 
ly wages and number of hours worked. 
Annual earnings also allow evaluators to 
compare the wages of those who work as 
employees with those who are self- 
employed. Also, it is often interesting to 
examine and compare the earnings of 
those who are employed in the occupa- 
tions for which they were trained with 
those who are working in occupations 
unrelated to their training. 

The degree of employee satisfaction with 
work is classified here as an economic 
outcome since it is often reflected in the 
productivity of an individual. Although it 
is also influenced by many factors beyond 
the control of vocational education, it can 
reflect the degree to which a student's 
program developed a realistic picture of 
the workplace. 

Similarly, employer satisfaction with 
employees is classified as an economic 
outcome since it can be a reflection of a 
company's success in the marketplace. 



Again, this factor is influenced by many 
factors beyond how well the employer is 
satisfied with the educational preparation 
individual employees have received. 
Therefore, evaluators need to interpret 
the results obtained for this outcome and 
the previous one with caution. 



Educational Outcomes 

Legislators and policy makers are moving 
more toward accountability measures tied 
to the instructional outcomes of educa- 
tional agencies. This reflects quite a 
departure from the institutional or process 
evaluations that have been used in the 
past. This growing concern for educa- 
tional outcomes is being reflected in revi- 
sions to statewide approaches to voca- 
tional education evaluation in states such 
as Illinois, Minnesota, and Virginia. Pos- 
sible educational outcomes include- 



The development of academic skills in 
such areas as mathematics and English is 
an outcome that more than 86 percent of 
the public perceives as important for all 
students (Elam 1990). The high level of 
importance placed on these subjects indi- 
cates that vocational teachers should also 
emphasize the development of these skills 
in their courses by showing students how 
these skills can be applied. 

Higher-order thinking skills require stu- 
dents to do more than simply recall or 



• Academic skills 

• Higher-order thinking skills 

• Knowledge of the world of work 

• Occupational sk ,# Us 

• School attendance and dropout 
rates 

• Continuing education rates 

• Students' satisfaction with 
education 



understand information. The develop- 
ment of these higher-order thinking skills 
requires students to apply, analyze, synthe- 
size, and/or evaluate information (Bloom 
1956) in solving problems. If teachers are 
to develop these type *, of skills, the objec- 
tives, instructional activities, and evalua- 
tion processes should reflect this type of 
emphasis. 

Another educational outco ' is knowl- 
edge of the world of work *>s Cetron 
and Davies (1989) predictei * the next 
d:cade, new jobs will appear and obsolete 
jobs will disappear faster tb? ever before, 
individuals will need to know the types of 
jobs mat will be available to them as the 
work force changes. Among the types of 
jobs that Cetron and Davis predicted as 
good opportunities are computers and 
data processing, telecommunications, 
health care, biotechnology, advanced 
materials, hazardous-waste disposal and 
pollution control, education and voca- 
tional training, and personal and business 
services. 

Occupational skills have been the major 
outcome of interest to many vocational 
educators. Much of the evidence for this 
outcome has been provided through the 
use of competency or performance testing. 
Though the public believes that vocational 
education should be required for all stu- 
dents not planning to go on to college 
(Elam 1990), evidence on student occupa- 
tional achievement is still needed. 

School attendance and dropout rates are 
particularly important outcomes. If the 
country is to increase the educational level 
of students, it is imperative that school 
attendance rates be increased and dropout 
rates be decreased. These rates should be 
examined for differences among racial/ 
ethnic groups, given the wide disparity in 
these rates across groups. 



The continuing education rates for stu- 
dents represent a recent outcome for 
vocational education. In a national longi- 
tudinal study of more than 6,000 individ- 
uals, Laughlin (1986) reported that 62 
percent of vocational education initially 
enrolled in some type of postsecondary 
education and training. Of those who 
enrolled, 62 percent completed the pro- 
gram or were still enrolled at the time of 
the follow-up. 



Psychosocial Outcomes 

The psychosocial outcomes of vocational 
education relate primarily to the personal 
development of the individual student. 
These outcomes are often the resuit of 
experiences in vocational education stu- 
dent organizations. Some of these out- 
comes are difficult to measure, and indi- 
cators of their effectiveness are not readily 
available. These outcomes include- 



• Aspirations 

• Attitudes and values 

• Self-esteem 

• Citizenship 

• Leadership 



The aspirations of students refer to their 
desires and ambitions. Many students do 
not have role models to use as examples 
in setting personal goals and selecting 
potential career areas. Therefore, the 
experiential learning provided by voca- 
tional education provides students with 
real-life examples to use in planning their 
future. Sometimes the personal contacts 
provide the opportunity for students to 
identify role models that they wish to 
emulate. 

Employers often criticize the attitudes and 
values that individuals bring to the work- 
place. Teachers and administrators have 
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the opportunity to influence students' atti- 
tudes and values by the personal examples 
they give to students. This outcome can 
also be influenced through the student's 
occupational experience program and the 
vocational student organization. 

Self-esteem refers to the belief students 
have in themselves. It also refers to their 
self-respect. This outcome can be influ- 
enced by how individuals interact with 
students. Positive learning environments 
need to be established by teachers, admin- 
istrators, occupational experience supervis- 
ors, and other students and co-workers. 

The outcome of citizenship refers to how 
students demonstrate their membership in 



society by acts such as voting and partici- 
pation in community organizations. The 
vocational student organizations provide 
microcosms in which to practice the du- 
ties, rights, and privileges related to this 
outcome. 

Leadership skills will be needed through- 
out a student's life. These skills will 
determine how students direct, command, 
or guide their colleagues or an activity. 
Excellent opportunities for students to 
develop these skills, as well as citizenship, 
are provHed by the vocational student 
organization. 
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USING THE EVALUATION FRAMEWORK 



This section describes how to use the 
framework in evaluating vocational educa- 
tion programs. As mentioned earlier, the 
framework is not a rigid model, but a 
guide for thinking through evaluations of 
vocational education programs. As such, 
the framework is helpful in delineating 
and designing program evaluations. 

To use this framework, it must be adapted 
to the situation. First, it will be neces- 
sary to decide the type, purpose, and 
extent of the evaluation. Then, it will be 
necessary to set priorities on the most 
important information to be collected. 
Once these determinations have been 
made, data can be collected and the infor- 
mation analyzed and interpreted. Finally, 
a report can be prepared presenting the 
results of the evaluation. Each of these 
steps is described here. 

Determine the Type, Purpose, and 
Extent of the Evaluation 

Earlier, a distinction was made between 
two types of evaluation: formative and 
summative. Formative evaluation occurs 
throughout the operation of a program to 
determine how it might be revised or 
improved. Summative evaluation is 
conducted at the end of a program to 
provide evidence on the effectiveness of 
the program and is used primarily for 
accountability purposes. The type of 
evaluation to be conducted is affected by 
whoever requested the evaluation. 
Vocational program operators most often 
are interested in formative evaluation, 
whereas program funders and sponsors 



usually are interested in summative 
evaluation. Therefore, it is essential to 
determine the type of evaluation that is 
desired. Will the primary purpose of the 
information be to improve the operation 
of the program (formative) or to provide 
accountability information (sumTnaiive)? 

Several purposes for conducting evalua- 
tions exist (Brinkerbotf, Brethower, 
Hluchyj, and Nowakovski 1983a). Among 
the major purposes that an evaluation 
might address are the following: 

• To find out if there is a need for the 
program 

• To determine if the goals and 
objectives of a program have been 
met 

• To decide if the ways in which a 
program operates are adequate 

• To see if a program operates in the 
way it was intended 

• To determine the outcomes of a 
program 

Finally, a decision must be made regard- 
ing the extent of the evaluation. Is it to 
involve all the vocational programs in an 
educational organization? Is it to be 
selective and address only vocational pro- 
grams that are experiencing problems in 
reaching their objectives or are not 
achieving the desired level of performance 
standards? Should information be col- 
lected from all the students? Should 
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information be collected only from at-risk 
students? 

The decisions regarding the type, purpose, 
and extent of evaluation are very impor- 
tant if useful and objective evaluations are 
to be designed and conducted. A design 
team of seven to nine individuals repre- 
senting local advisory groups, teachers, 
and administrators can be used in making 
these decisions. The majority of this team 
should be composed of people outside the 
educational organization if the evaluation 
is to be viewed as a credible assessment. 



Select the Components 
to Be Included 

Once the type, purpose, and extent of the 
evaluation have been decided, the com- 
ponents of the evaluation must be chosen 
from the framework. The current empha- 
sis on documenting the results of educa- 
tional programs suggest that the compon- 
ents of vocational education outcomes 
provide an appropriate place to start. 

The Carl D. Perkins Vocational and 
Applied Technology Education Act of 
1990 states that all vocational education 
programs are to evaluate their programs 
using measures of iearning and compe- 
tency gains including student progress in 
the achievement of basic and more 
advanced academic skills. The component 
in the framework that addresses this area 
is academic skills. The Perkins Act also 
calls for one or more measures of perfor- 
mance from outcomes such as occupation- 
al skills, retention in school, placement 
into additional training or education, and 
placement into the military service or 
employment. These outcomes are includ- 
ed in the framework as occupational skills, 
school attendance rates, continuing educa- 
tion rates, training-related placement, and 
employment rates. 



Some states may have other types of out- 
comes that they wish to emphasize in their 
vocational education programs. Local 
educational agencies may also have the 
same or other outcomes of interest to 
them. All these factors need to be con- 
sidered when selecting the type of out- 
comes used to focus the evaluation of 
vocational education programs. 

The next step is to select components 
from the framework that relate to voca- 
tional education processes and needs for 
vocational education. These components 
should be those that most likely influence 
the vocational education outcomes that 
were selected. This information will be 
needed to help explain or interpret the 
findings f the evaluation. 

It should not be surprising to find that 
there are more components of the evalua- 
tion than can be included. Therefore, 
these components must be placed in order 
of priority based on the type and purpose 
of evaluation that was selected in the 
previous step. Once this priority has been 
established, the selection can be based on 
the resources available. 



Identify the Sources and Types of 
Information to Be Collected 

Once the components to be included in 
the evaluation are selected, the type of 
data or information that it will hi neces- 
sary to collect must be identified. A sam- 
ple listing of potential sources of informa- 
tion for each component is presented in 
figure 3. A wide range of potential 
sources of information-both qualitative 
and quantitative-exists. Whenever possi- 
ble it is desirable to obtain more than one 
type of information about a component to 
be included in the evaluation of vocational 
education programs. 
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Component 



Potential Sources of Information 



Academic Skills 



Educational Achievement by Grade and Age on the Iowa 
Tests of Basic Skills or California Achievement Tests 



Occupational Skills 



Educational Competence 
of Staff 



National Occupational Competency Tests for an 
occupation 

College Transcripts 

National Teachers Examination 



Demographic Charr teristics 
of Students 



Individual Student Records 
Interviews with Students 



Goals and Objectives of 
Program 



Course of Study 
Interviews with Teachers 



Instructional Content 



Lesson Plans 
Observation of Classes 



Instructional Delivery 
Methods 



Observation of Classes 
Interviews with Teachers 



Teaching Styles 



Observation of Teachers 
Survey of Teaching Styles 



Figure 3. A sample listing of potential sources of evaluation information 



Collect the Required Information 

Collecting information is an important 
step in conducting an evaluation of voca- 
tional education programs. This informa- 
tion will provide the evidence to be used 
in judging the value of the pxogram. Re- 
member that the evaluation will only be 
as good as the quality of information col- 
lected. According to Worthen and 
Sanders (1987)- 

information is situational and 
changes with every evaluation. 
Similarly, the methods evaluators 
must use to collect information 
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also must change. Sometimes 
alternative data-collection meth- 
ods are available, and the evalua- 
tor must make a choice, consider- 
ing cost, precision (reliability), 
stability, relevance, validity of 
measurements, feasibility, political 
advisability, and acceptability to 
various audiences. The evaluator 
must be ready to use whichever 
method or methods appear most 
appropriate. And T>eing pre- 
pared" usually means having prior 
experience. We advise evaluators 
to gain experience with any meth- 
ods that may be new to them- 
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and that means actually using the 
methods, not just reading about 
them. (pp. 298-299) 

Usually it is necessary to establish agree- 
ments with those whose assistance will be 
needed in collecting evaluation informa- 
tion. Large schools often have very de- 
tailed procedures that must be followed 
In these efforts. Approval for accessing 
the required information may be needed. 



Analyze and Interpret the 
Information Collected 

The process ol analyzing and interpreting 
the information collected is also an 
important task. It is likely that evaluation 
efforts will collect a large amount of 
information. Somehow this information 
must be assembled cohesively so that 
others can comprehend the findings. 

Analysis of ^ata is one way that informa- 
tion can be reduced to a meaningful level. 
Often, measures of central tendency (that 
is, medians, means, modes) are used to 
summarize quantitative data. Other 
times, ranges and correlations may need 
to be used to indicate the diversity or 
similarity of data. It may even be 
necessary f o use statistics such as chi- 
square, t-tests, and analysis of variance to 
interpret the information. 

The analysis of qualitative information 
requires the evaluator to be involved in 
an ongoing process of observation and 
reflection, drawing conclusions and testing 
these conclusions as additional informa- 



tion becomes available. Miles and Huber- 
man (1984) provided an excellent discus- 
sion regarding the analysis of qualitative 
data. 



Prepare an Evaluation Report 

The results of the evaluation need to be 
reported to the stakeholders (those direct- 
ly affected by the evaluation results) of 
the vocational education program. These 
reports can be used for a variety of pur- 
poses besides determining the worth of a 
program. Other uses include image build- 
ing, education, and public relations. Be- 
sides considering the purpose an evalua- 
tion report is to serve, evaluators need to 
consider the audience for the report. The 
purpose and the intended audience will 
dictate the form of the evaluation report. 
Worthen and Sanders (1987) suggested 
the following types of evaluation reports: 
written reports, photo essays, audiotaped 
reports, slide-tape presentations, filmed or 
videotaped reports, multimedia presenta- 
tions, dialogues/testimonies, hearings or 
mock trials, product displays, simulations, 
scenarios, portrayals, case studies, graphs 
and charts, test score summaries, and 
questions/answers (pp. 344-345). 



Concluding Remark 

This paper is not intended to be a com- 
plete handbook for conducting evalua- 
tions. Handbooks and guidelines of this 
type are available by such authors as 
Brinkerhoff et al. (1983a,b), Herman 
(1989), and Worthen and Sanders (1987). 
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